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Foe  a  substantial  number  of  companies  competing  in  a  wide  range  of 
industries  today,  information  technology  has  become  a  critical  business 
variable.   Expectations  for  its  strategic  impact  have  heightened  in  many 
sectors,  fueled  largely  by  consultants,  academics,  and  managers  who  have 
documented  on  a  case  by  case  basis  the  successful  and  failed  attempts  to 
couple  I.T.  investments  more  tightly  to  core  business  objectives.   In 
spite  of  these  efforts,  however,  some  senior  executives  today  remain 
skeptical  about  their  ability  to  capture  any  strategic  benefits  from  such 
investments. 

During  the  past  eighteen  months,  we  have  interviewed  71  chief  executive 
officers  in  ten  different  industrial  sectors  to  learn  more  about  their 
beliefs  and  assumptions  about  information  technology.   (1)   From  these 
interviews,  we  have  observed  that  top  managers  are  concerned  about  two 
key  dimensions  of  I.T.   The  first  relates  to  their  assumptions  about 
payoff:   the  scope  and  magnitude  of  expected  benefits  from  I.T.   The 
second  relates  to  their  own  personal  calculations  of  the  likelihood  that 
their  firms  will  realize  some  or  all  of  these  benefits.  One  unexpected 
finding  from  our  interviews  is  that  many  of  these  CEOs,  including  those 
whom  we  identified  as  "Utopians"  because  they  are  so  positive  on  both 
dimensions,  are  less  than  completely  satisfied  with  the  methods  used  in 
their  organizations  for  assessing  I.T,  performance.   The  CEO  of  one  of 
the  20  high  technology  firms  we  interviewed  asked  us: 

What  benefits  does  I.T.  provide  to  the  organization? 
Do  people  really  need  it  as  much  as  they  say  they  do? 
We  are  able  to  cost  justify  the  equipment,  but  is  it 
really  making  our  people  more  effective? 

The  chairman  of  a  large  insurance  corporation  confided: 
It's  hard  to  know  what  produces  competitive 


advantage.   MIS?   I  don't  know  how  to  measure  its 
impact . 


And  the  CEO  of  a  mid-sized  company  admitted: 


There  are  a  few  specific  cases  where  /ou  can  tell  — 
like  with  robots.   You  can  see  what  they  cost,  measure 
their  productivity,  and  calculate  if  they  are 
providing  advantage.   But  for  the  most  part,  computing 
is  not  that  way.   I.S.?   I  don't  know  how  to  measure 
its  impact. 


These  top  managers  have  three  fundamental  questions  when  it  comes  to 
assessing  I.r.  performance: 

1.  How  well  are  we  doing  the  things  we  are  doing  now? 

2.  Are  we  doing  the  right  things? 

3.  Are  we  fit  to  compete  in  the  future? 


The  Expert  Panel 

Concerns  such  as  those  expressed  oy  these  CEOs  provided  the  impetus  for 
us  to  interview  thirteen  experts  in  the  managertient  of  inforToation 
technolog/  in  order  to  elicit  their  views  on  how  to  evaluate  I.T. 
performance.   We  chose  professors  who  teach  information  technology 
courses  at  two  leading  academic  institutions  and  asked  them  to  recommend 
names  of  other  experts  for  whose  opinions  they  had  high  regard.   From 
this  list  of  18  experts,  in  addition  to  the  previously  mentioned 
academics,  we  selected  five  consultants  of  information  technology,  three 
managers  of  information  systems  from  different  Fortune  50  corporations, 
and  a  researcher  actively  studying  the  impact  of  information  technology 


on  organizational  pecEormance. 

Our  quer/  to  each  expert  was  a  simple  one:   if  asked  by  a  CEO  to  assess  a 
firm's  I.T.  performance,  how  would  you  do  it?  What  questions  should  be 
asked  and  what  information  would  you  need? 


One  General  Conclusion 

The  panel  study  generated  two  kinds  of  information:   an  extensive  set  of 

indicators  of  effectiveness  and  seven  distinct  assessment  methods.   In 

summary,  66  different  indicators  of  I.T.  performance  were  discussed, 

which  raises  a  question  about  whether  managers  should  avoid  relying  on 

just  one  assessment  method  or  technique  and,  instead,  should  employ 

several  methods  when  evaluating  their  I.T.  investments  and  plans.  One 

expert  was  emphatic  about  this  advice: 

There  is  not  now,  nor  will  there  ever  be,  a 
convergence  of  opinion  among  experts  regarding  how  I.T 
effectiveness  is  to  be  assessed.   There  are  no  more 
clear  indicators  of  effectiveness  than  there  are  [of] 
goodness  or  badness  of  companies.   In  different 
environments,  different  metrics  give  people  comfort. 
I  wouldn't  expect  people  to  come  up  with  a  single 
metric  of  effectiveness. 

(Professor ) 

Based  on  the  results  of  this  panel  study,  it  is  reasonable  to  conclude 
that  a  single  index  of  I.T.  effectiveness  that  has  been  widely  adopted 
does  not  exist.   One  explanation  for  why  so  many  different  indicators  of 
effectiveness  are  used  is  that  the  management  of  information  technology 
is  in  a  state  of  transition  and,  consequently,  the  managerial  control 
systems  that  help  executives  keep  score  in  this  area  and  determine  how 


Eit  theic  compan/  is  £oc  action  also  are  in  a  state  of  evolution. 

Three  oE  the  methods  discussed  by  the  experts  are  familiar  and 
established  approaches  and  primarily  reflect  what  has  been  the  dominant 
managerial  vision  of  I.T.  —  the  automate  perspective. 

However,  we  also  discuss  four  other  assessment  methods  that  at  present 
seem  underdeveloped  because  they  reflect  a  new  and  emerging  view  of  I.T., 
one  that  involves  the  applications  of  I.T.  to  new  domains  of  work  and 
tasks.   Panel  members  expressed  agreement  about  the  need  for  new  forms  of 
control.   Several  expressed  deep  concerns  about  the  way  that  many  firms 
use  traditional  cost/benefit  analyses: 


Everybody  does  cost/benefit  analysis  on  projects. 
Most  of  them  are  fictional.   The  saddest  part  is  that 
it  is  not  just  the  benefits  that  are  fictional,  but 
the  costs  as  well.   They  don't  begin  to  assess  what 
the  real  costs  of  the  systems  ace.  You  do 
cost/benefit  analysis  to  get  money. 

We  have  trouble  convincing  ourselves  of  value  in 
business  terms.  We  cost  justify  new  systems  on  the 
basis  of  lies,  and  having  justified  [them]  on  the 
basis  of  lies  and  overstatement,  we  then  don't  measure 
the  true  business  value. 

(Professor) 


Experts  were  in  general  agreement  that  professionals  do  a  good 
job  of  assessing  performance  for  existing  applications  that 
cover  current  business  processes,  but  are  not  particularly  good 
at  assessing  capabilities  for  transforming  current  business 
processes  into  those  which  will  position  a  firm  for  future 
competitive  action. 


Developing  Assessment  Concepts 

The  pucpose  o£  this  papec  is  to  provide  a  conceptual  framework 
determining  what  are  appropriate  types  o£  control  for  different  executive 
visions  or  expectations  of  I.r.  performance.   The  framework  is  an 
outgrowth  of  these  expert  views  and  is  grounded  in  the  experience  of  this 
particular  set  of  people. 

The  four  ke/  elements  of  the  framework  are  displayed  below. 

Figure  1.   DETERMINANTS  OF  TYPES  OF  I.T.  CONTROL 
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We  discuss  here  seven  different  assessment  methods  recommended  by  this 
expert  panel  and  analyze  the  usefulness  of  each  for  assessing  performance 
for  three  distinctly  different  but  broadly  understood  visions  of  I.T. 
capability.  We  refer  to  these  as  a  vision  to  automate,  a  vision  to 
inf ormate  (2)  and  a  vision  to  transform  the  organization.  (3)   The  vision 
to  automate  is  the  established  vision;  visions  to  informate  and  cransform 


are  emerging  and,  in  most  organizations  today,  are  represented  more  as 


expectation  than  as  reality.   We  have  taken  this  approach  because  we 
suspect  that  the  sense  of  frustration  and  dissatisfaction  that  man/ 
executives  experience  when  assessing  I.T.  performance  results  from 
emplo/ing  justification  and  evaluation  methods  which  are  appropriate  for 
the  automating  vision,  but  which  do  not  reall/  apply  to  monitoring 
progress  on  the  other  two  visions.   The  unfortunate  consequence  of  such  a 
situation  is  that  decisionmakers  become  trapped  by  a  planning  and  control 
system  that  utilizes  measures  with  little  validity  for  assessing  progress 
toward  informating  or  transforming  organizations. 

The  vision  to  automate  involves  the  application  of  computers, 
communications  and  software  to  existing  tasks  and  processes.   It  presumes 
detailed  knowledge  of  the  transformation  process,  because  a  vision  to 
automate  involves  substituting  the  computer  program  for  an  already 
routine  manual  process.   This  vision  becomes  real  when  applications  are 
developed  for  uniform  transactions  and  standardized,  structured 
workflows.  With  goals  of  improving  efficiency  and  productivity,  of 
standardizing  operations,  and  of  modernization,  this  vision,  for  the  last 
25  years,  has  provided  the  primary  justification  for  managers  to  invest 
billions  of  dollars  in  computers,  software,  and  telecommunications 
equipment.  Guided  by  this  automating  perspective  of  I.T.'s  value, 
corporate  managers  justify  investments  based  on  expectations  of  reducing 
errors  in  the  workflow,  decreasing  labor  costs,  and  increasing 
productivity. 

A  vision  to  informate  places  a  unique  emphasis  on  expanding  the 
intellectual  content  of  work  and  on  distributing  knowledge  about 
administrative  and  production  processes  at  all  levels  of  the 


organization.   For  example,  scanner  devices  in  supermarkets  automate  the 
checkout  process  while  simultaneousl/  generating  data  that  can  be  used 
for  inventor/  control,  scheduling  of  deliveries  and  market  anal/sis.  (4) 
executive  aspirations  are  to  generate  new  streams  of  information  and 
expand  anal/tical  capabilities,  to  emphasize  more  comprehensive  jobs,  and 
to  change  authority  relationships  between  managers  and  employees. 

Only  recently  has  the  vision  to  transform  the  organization  around  the 
unique  strengths  and  characteristics  of  I.r.  gained  acceptance  and 
understanding.  This  vision  incorporates  a  significant  element  of  planned 
change  that  ordinarily  is  not  found  in  organizations  that  are 
automating.   This  view  defines  information  technology  as  an  instrument 
for  changing  the  factors  of  competition,  and  it  targets  flexibility  and 
change  as  essential  elements  of  all  business  and  functional  action 
programs.   Such  a  vision  is  derived  from  an  assumption  that 
discontinuities  in  market  equilibrium  will  provide  strategic 
opportunities,  and  that  only  by  optimal  utilization  of  I.r.  will  these 
opportunities  be  realized.   One  expert  on  the  panel  used  investment 
banking  to  illustrate  the  need  for  such  a  vision  in  some  businesses  today: 


No  industry  has  moved  more  sharply  or  more  completely 
from  I.r.  being  just  a  support  to  the  business  to  it 
being  a  critical  factor  of  competition.   In  1980, 
computer  operations  in  investment  banking  firms  were 
hidden  down  in  the  backwaters  of  the  companies  — 
supremely  irrelevant,  vastly  overpaid,  low  level  kinds 
of  people.   In  1987,  there  they  are,  right  out  on  the 
sharp  tough  point  in  terms  of  managing  the  process, 
the  cultural  change  and  shift  in  perspective.   And 
it's  been  tough.   The  oroolem  that  has  consumed  them, 
unhappily,  is  managing  the  cultural  transformation 
from  the  supremely  irrelevant  to  the  extremely 
relevant. 

(Professor ) 


Managers  have  had  far  more  experience  in  implementing  visions  to 
aatomate.   Visions  to  transform  and  inEormate  presume  an  acceptance  o£ 
change  without  knowledge  of  what  the  new  form  of  organization  and 
competition  will  be,  making  it  particularly  difficult  to  assess 
performance. 

We  define  effectiveness  as  the  ability  to  bring  about  results;  to  show 
how  performance  is  related  to  a  set  of  goals  or  standards.   Defined  as 
such,  any  assessment  process  should  focus  on  the  attainment  of  goals  and, 
depending  on  the  outcomes,  trigger  responses  which  reflect  learning  and 
adaptation. 

An  index  of  I.r.  effectiveness,  then,  requires  two  preference  orderings: 
1)  someone  must  choose  and  weigh  the  goals  or  expectations  to  be 
evaluated  and;  2)  someone  must  designate  what  the  standards  or  measures 
are  for  determining  the  extent  to  which  these  expectations  have  been 
achieved.  One  major  dilemma,  then,  that  has  to  be  resolved  in  any 
assessment  is  to  decide  whose  preferences  or  goals  should  form  the  basis 
of  the  assessment.  There  may  be  significant  differences  in  the  goals 
among  corporate,  business  and  functional  managers,  or  goals  may  not  be 
clearly  stated,  revealing  a  weak  or  even  conflicting  preferential 
ordering. 

A  good  assessment  method  should  also  take  into  account  the  constraints 
which  limit  performance.   Cack  of  resources,  the  stage  of  organizational 
development,  and  a  range  of  environmental  factors  may  explain  poor 
performance;  management's  explicit  goals  for  I.r.  may  not  be  realistic  in 
context  of  such  constraints. 


The  expert  interviews  revealed  two  basic  types  of  variables  used  in 
assessment:  output  variables,  such  as  sales,  profit,  growth,  market 
share,  MIPS  pec  transaction,  and  hardware  cost  performance;  and 
behavioral  variables,  such  as  actions,  conduct,  how  tasks  are  performed, 
managers'  abilities  to  influence  and  motivate,  attitudes,  and  knowledge. 
This  raises  an  important  question.  Under  what  conditions  is  one  type  of 
control  variable  feasible  or  preferable  over  the  other?  How  should 
managers  decide  which  control  variable  to  use? 

To  answer  this  question,  we  build  on  a  body  of  research  (5)  that  identifies 
the  conditions  for  determining  what  are  appropriate  control  systems.  One 
determinant  is  the  knowledge  of  the  transformation  process;  the  extent  to 
which  rules  of  behavior  and  process  can  be  specified.  Or,  stated  differently, 
this  dimension  is  an  estimation  of  the  extent  to  which  we  can  program  a  series 
of  tasks  or  a  process.   The  second  condition  is  the  extent  to  which  we  have 
reliable  measures  of  output  or  ambiguous  standards  of  performance.  Using 
extreme  values  for  these  two  dimensions,  we  see  from  Figure  2  that  four 
assessment  choices  are  possible: 


Figure   2. 
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One  assumption  of  organizational  contcol  is  thc'.i  it  is  f-,ii   e  to 
measure  desired  performance  with  reasonable  accuracy.   Su.-  .  in  assumption 
is  satisfied  by  the  existence  of  an  efficiency  test.   In  situations  where 
the  task  environment  is  stable  and  we  have  perfect  knowledge  of  tasks  and 
rules  of  behavior,  as  well  as  reliable  and  precise  standards  of  output, 
it  is  possible  to  determine  whether  a  result  has  been  produced  with  least 
cost:  or,  given  a  fixed  amount  of  resource,  whether  the  greatest  result 
has  been  achieved.   The  results  of  an  efficiency  test  ace  indisputable. 
Cost,  quantity,  and  quality  are  the  key  variables  of  such  tests.   An 
example  might  be  the  number  of  lines  of  code  produced  d/  a  programmer, 
the  number  of  MIPS  per  transaction,  or  the  number  of  checks  processed 
during  a  given  period  of  time. 

But  efficiency  tests  do  not  answer  the  institutional  question  of 
importance  to  executives  at  the  top  of  organizations:   what  has  the 
investment  in  I.T.  or  the  function  contributed  to  the  organization's 
overall  performance?   for  this,  managers  will  use  an  instrumental  test 
which  answers  whether  a  desired  state  of  affairs  has  been  achieved,  and 
whether  they  will  use  simple,  quantifiable  measures.   For  situations  like 
this,  in  which  the  task  environment  is  unstable,  and  knowledge  about  the 
transformation  process  imperfect,  efficiency  tests  are  not  availaole. 

The  problem  for  managers  under  these  conditions  is  that  exogenous 
variables  are  affecting  technical  results,  or  are  operating  in 
unpredictable  ways,  leaving  them  without  perfect  knowledge  of  how  the 
tasks  are  or  should  be  organized.  Under  such  conditions,  managers  will 
rely  on  subjective,  but  measurable,  output  variables.   An  example  might 
be  the  introduction  of  workstations  to  sales  representatives  who  work 
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independent!/  and  whose  work  processes  are  unstructured  and  highly 
variable.   In  such  cases,  I.T.  performance  will  be  measured  using  an 
output  variable,  such  as  individual  sales  revenue.   Another  example  is 
those  situations  in  which  top  executives  who  have  very  incomplete 
knowledge  of  how  the  I.T.  resource  is  being  applied  or  delivered  in 
behavioral  terms,  rely  on  return  on  investment  as  a  measure  of 
performance. 

When  the  task  environment  is  stable  and  knowledge  of  the  transformation 
process  and  rules  of  behavior  are  perfect,  but  output  measures  cannot  be 
expressed  in  quantitative  terms,  managers  rely  on  social  tests  of 
performance,  usually  measured  as  the  level  of  confidence  expressed  by  the 
units  which  rely  on  the  unit  being  assessed.   In  other  words,  what  is 
being  assessed  is  the  extent  to  which  actual  behavior  and  process  conform 
to  what  managers  in  these  units  believe  should  be  done.   For  example, 
imagine  that  the  marketing  department  of  a  large  company  relies  on  a 
central  I.S.  department  for  its  applications  development  support.   It  is 
the  marketing  users  who  have  the  complete  knowledge  of  their  work  process 
and  therefore  are  most  capable  of  assessing  the  value  or  performance  of 
the  services  provided  by  the  I.S.  unit.   Here  it  would  be  problematical 
to  identify  reliable  output  measures  for  assessing  improved  customer 
relationships  or  more  sophisticated  marketing  knowledge  (an  informate 
benefit),  and  so  managers  rely  on  statements  of  confidence  made  by  users. 

In  dynamic  task  environments,  when  knowledge  of  the  transformation 
process  is  imperfect  and  output  measurability  low,  managers  will  rely  on 
two  more  types  of  social  tests;   one  that  involves  comparison  with 
comparable  organizations,  the  other  a  test  of  ritual  and  ceremony  which 
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presumes  that  an  underlying  set  o£  values,  attitudes  and  motivations  on 
the  pact  of  key  selected  individuals  will  lead  to  success.   For 
situations  in  which  uncertainty  about  the  task  environment  is  very  high, 
and  performance  measucability  extremely  low  —  as  when  executives  ask 
themselves,  "Are  we  fit  to  compete  in  the  future?"  —  managers  tend  to 
rely  on  outside  experts  to  judge  their  I.r.  performance,  or  to  recruit 
individuals  with  the  professional  values  they  believe  will  assure  their 
ability  to  compete  in  the  future.   One  example  might  be  those  firms  in 
which  a  distinctly  different  type  of  manager  than  the  traditionally 
trained  systems  manager  has  been  put  in  charge  of  the  I.T.  function.   By 
appointing  a  marketer  or  a  business  manager  to  head  this  function,  a 
message  is  being  communicated  that  past  performance  will  not  necessarily 
assure  future  success  and  that  changes  will  have  to  be  made. 

With  this  conceptual  framework,  we  now  are  able  to  do  several  things. 
First,  we  can  distinguish  different  assessment  methods  based  on  their 
reliance  on  behavioral  or  output  measures.   Secondly,  we  can  determine 
the  appropriateness  of  specific  assessment  methods  for  different  visions 
and  applications  of  I.T.   Third,  we  can  determine  whether  a  firm  is 
utilizing  controls  systems  that  are  consistent  with  executives' 
expectations  for  I.T. 

In  order  to  do  this,  each  of  the  thirteen  expert  interviews  was  analyzed 
in  several  ways.   Each  interviewed  for  all  indicators,  yielding  a  total 
of  95  indicators.   Duplicate  variables  were  removed  from  the  summary 
list,  leaving  66  indicators,  each  of  which  then  was  coded  as  to  its  type 
—  output  or  behavior.   The  original  interviews  were  analyzed  again,  this 
time  to  document  any  self-contained  methods.   Seven  different  methods 
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emerged  from  this  analysis,  and  each  indicator  was  assigned  to  one  of 
these  methods.   Figure  3  lists  these.   It  should  be  noted  that  for  many 
of  the  indicators  from  the  list  of  66,  measures  were  not  specified;  and 
so  the  question  of  whether  operational  procedures  exist  for  all  of  these 
cannot  be  answered  by  the  data  presented  here. 

In  the  following  sections  of  the  paper,  we  describe  each  of  the  seven 
methods  and  analyze  their  usefulness  for  different  contingencies, 
building  our  analysis  upon  the  control  framework.   The  labels  used  to 
describe  these  methods  are  my  own.   While  no  claim  can  be  made  that  these 
seven  methods  of  assessment  exhaust  all  the  possible  approaches  which 
firms  employ,  they  do  incorporate  all  the  key  variables  recommended  by 
this  particular  panel  of  experts  who  jointly  encompass  a  broad  range  of 
corporate  experience  and  practice. 

Figure  3.   SEVEN  I.T.  ASSSSSMEMT  METHODS 

Productivity  Assessment 

User  Utility  Assessment 

Value  Chain  Assessment 

Competitive  Performance  Assessment 

Business  Alignment  Assessment 

Investment  Targeting  Assessment 

Management  Vision  Assessment 
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In  the  last  section  of  the  paper,  conclusions  are  made  regarding  the 
appropriateness  of  these  methods  in  context  of  the  three  managerial 
visions  of  information  technology. 


Assessing  Productivity 

The  first  approach  focuses  on  determining  the  efficiency  with  which  I.T. 
resources  are  expended.   There  is  some  divergence  in  experts'  opinions 
regarding  professional  ability  to  evaluate  I.T.  productivity  with  any 
reliability  and  precision;  but,  if  such  measures  exist  and  are  used,  this 
type  of  assessment  should  provide  managers  with  enough  information  to 
answer  the  question  "Are  we  doing  the  things  we  are  doing  well?"   The 
measures  predominantly  used  in  assessing  I.T  productivity  include  ■ 
assessments  of  the  volume  of  work  or  number  of  transactions  processed; 
e.g.  code,  loans,  checks,  telephone  calls,  claims,  etc.   Systems  measures 
that  were  mentioned  include  on-line  systems  performance,  systems 
development  productivity  —  or  number  of  programs  per  programmer  — 
hardware  costs  versus  performance,  and  line  of  code  per  programmer. 

Cost,  quantit/,  and  quality  are  key  variables  in  such  assessments,  and 
what  determines  the  appropriateness  of  using  this  method  are  whether:  1) 
perfect  knowledge  of  the  transformation  process  exists;  and  2)  precise, 
quantifiable  output  measures  exist.   Several  experts  identified  "quality 
of  data,"  "quality  of  the  technology  architecture,"  and  "quality  of  the 
I.T.  staff"  as  important  indicators;  but  without  precise  empirical 
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measaces  Eor  such  "outputs,"  these  can  only  oe  evaluated  through 
instrumental  or  social  tests. 

Productivity  is  measured  best  when  a  business'  or  function's  output  is 
fairly  homogeneous  and  does  not  reflect  much  change  over  time.   Kilowatt 
hours  of  electricity  and  numbers  of  telephone  calls  are  reliable  measures 
of  output;  but  for  functions  or  businesses  in  which  output  is  an 
intangible  service  of  uncertain  or  variable  quality,  or  where  products 
change,  productivity  measurement  is  problematical.   This  explains  why 
managers  seek  other  means  for  assessing  performance,  such  as  user  utility 
or  value  chain  assessments. 


Assessing  User  Utility 

A  second  method  often  used  is  the  assessment  of  user  utility.   Different 
approaches  may  be  used  for  determining  whether  customers  and  clients  for 
I.T.  services  are  satisfied  and  perceive  value  in  these  services.   One 
expert  with  extensive  I.S.  experience  in  the  financial  services  sector 
commented: 


One  important  measure  is  user  feedback,  or  what  I  call 
the  noise  level.   If  the  noise  level  is  silent,  you're 
doing  a  pretty  good  job.   If  you  have  a  lot  of 
negatives  coming  from  users  who  are  complaining  about 
report  deliveries  that  are  late,  uptime  is  downtime, 
nothing  works  right,  costs  too  much  —  that's  when  you 
have  problems.   Bad  noise  level  is  bad  news.   Most 
senior  managers  don't  look  at  statistics.   They  look 
at  how  many  people  walk  through  the  door  complaining 
about  computers,  or  how  many  letters  are  written  by 
customers  complaining  about  computers.   It's  so 
simple.   Customers  who  get  a  statement  that's  wrong 
will  write  to  the  President. 

(Manager  ) 
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Some  l.T.  departments  conduct  periodic  surve/s  o£  their  users  to  find  out 
how  they  rate  the  utility  of  the  applications,  information,  and  other 
services  they  receive.   This  feedback  is  then  incorporated  into  long 
range  l.T.  plans.   One  expert  who  has  used  such  an  approach  explained, 
"The  results  of  a  biannual  user  survey  are  an  integral  part  of  our 
activities  and  plans.   This  is  the  proof  of  the  pudding.   How  effectively 
are  you  servicing  the  internal  customers?" 

Expert  opinion  is  divided,  however,  over  the  value  of  assessing  usee 
utility.   One  expert  warned: 


Is  l.T.  effeciveness  a  function  of  whether  the  users 
ace  happy  or  not?  Many  people  think  so.   I  say  no. 
That  is  a  dangerous  assumption.   Usage  as  a  measure  of 
effectiveness  implies  that  the  organizational  process 
and  structure  are  right. 

(Consultant) 


Another  expert  illustrated  this  problem  with  the  following: 


Take  a  contract  in  which  I  guarantee  to  deliver  95%  of 
your  critical  reports  on  time,  or  that  your  system 
will  be  up  93%  of  the  time.   Whatever  these  measures 
are,  they  are  agreed  to  in  advance,  documented  and 
reported  on  regularly.   The  only  problem  with  this  is 
that  no  matter  what  the  statistics  say,  the  only  thing 
users  remember  is  when  something  doesn't  work.   Even 
though  you  reached  the  98%  uptime  objective,  the  day 
it  was  down  happened  to  be  a  very  bad  day,  and  he  lost 
some  customer  business  because  of  it. 

(Manager) 


The  determinants  for  employing  this  type  of  control  system  are  a  perfect 
knowledge  of  the  transformation  process  and  low  output  measurability .   In 
situations  in  which  l.T.  is  applied  to  highly  programmable  tasks  or 
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processes,  using  this  form  of  a  social  test  is  appropriate. 
Assessing  Impact  on  the  Value  Chain 

The  third  method  for  assessing  I.r.  perfopcmance  is  a  value  chain 
assessment .   This  approach  attempts  to  quantif/  the  impact  of  I.T.  on 
functional  goals  by  using  performance  standards  established  by  functional 
managers  and  relies  primarily  on  output  variables. 

A  company's  value  chain  is  composed  of  nine  generic  technological  and 
economic  activities  as  described  below. 

Figure  4.   The  Value  Chain  (5) 
Th*  vahM  chain 
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Every  activity  in  the  value  chain  uses  information,  and  I.T.  may  have  an 
impact  on  the  linkages  between  these  value  activities  so  as  to  produce 
competitive  effects.   Value  chain  assessment  is  appropriate  for  analyzing 
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the  contcibation  or  impact  of  I.r.  on  the  attainment  o£  soeciEic  functional 
goals  which  are  measured  with  functional  output  variables. 
For  example,  manufacturing  might  measure  performance  using  such  variables 
as  "time  to  complete  designs,"  or  "overhead  cost  per  procurement  item";  or 
service  might  measure  "reductions  in  costs  of  deliver/,"  or  "increasing 
lead  time  in  financial  transactions."   If  used  in  conjunction  with  other 
methods,  such  as  user  utility  assessment  or  competitive  performance 
assessment,  decisionmakers  may  be  in  a  better  position  to  understand 
whether  I.T.  investments  are  tightly  linked  to  functional  needs  and 
performance. 

Similar  to  the  value  chain  assessment  is  the  critical  success  factors 
approach  (7),  by  which  managers  evaluate  the  extent  to  which  applications 
help  them  achieve  their  business  goals,  measured  with  output  variables  such 
as  sales,  revenue,  growth,  market  share,  profit,  earnings,  etc.   This  well 
known  approach  is  a  type  of  instrumental  test  and  can  be  used  at  the 
corporate  level  with  top  executive,  or  at  divisional  or  business  unit 
levels,  as  dictated  by  the  firm's  organizational  structure. 

The  important  distinction  between  the  value  chain  assessment  and  the  user 
utility  assessment  is  that  the  former  relies  on  output  variables,  while  the 
latter  method  is  appropriate  when  knowledge  of  the  transformation  process 
is  high  and  output  measurability  is  low. 


Assessing  Competitive  Performance 


For  many  executives,  their  primary  concern  is  whether  they  are  investing  in 
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the  right  projects  and  the  right  technology.   One  way  of  making  such  a 
determination  is  by  comparing  oneself  to  the  competition.   One  expert,  the 
vice  president  of  information  systems  for  a  large,  divisionalized 
manufacturer  of  industrial  products,  described  this  challenge: 

The  toughest  question  I  face  as  an  I.S.  director  with 
my  top  management  is  "Where  do  we  stand 
competitively?'  They  continue  to  feel  uncomfortable 
with  our  growth  factor.  We  are  the  fastest  growing 
function  in  terms  of  budget  and  people. 

(Manager ) 

There  are  several  variations  on  the  theme  of  competitive  performance 
assessment.  One  approach  calls  for  an  analysis  of  performance  on 
specific  infrastructure  components  —  e.g.  database,  telecommunications, 
hardware,  applications,  etc.  —  as  compared  to  that  of  companies  reputed 
to  be  performance  leaders  for  the  same  components,  regardless  of  whether 
or  not  the  companies  compete  in  the  same  marketplace.   The  practice  is 
described  by  an  expert  as  it  is  used  in  his  own  company: 


In  some  key  areas,  I  want  to  see  competitive 
benchmarks.   It's  comparing  ourselves  with  the  best  we 
can  find.   Let's  compare  our  data  centers  with  the 
best.   Let's  compare  our  telecommunications  network 
with  the  best.   Let's  compare  the  way  we  develop 
systems  with  the  best  we  can  find,  regardless  of  the 
industry. 

(Manager) 


A  second  approach  to  competitive  assessment  is  to  compare  an 
organization's  performance  on  specific  components  against  that  of  its 
immediate  competitors.   Such  evaluations  are  driven  by  managerial  goals 
to  maintain  an  I.r.  infrastructure  that  is  more  cost-effective  than  the 
competition.   One  major  difficulty  with  competitive  performance 
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assessments  is  that  the  measures  used  ma/  not  be  reliable  from  one  firm 
to  another.   Knowledge  about  the  competitors'  transformation  process  will 
be  imperfect  or  incomplete,  making  it  problematical  to  infer  with  an/ 
reliability  how  similar  or  comparable  performance  between  firms  really 
is.   One  expert  shared  his  experience  in  trying  to  find  useful  standard 
measures  for  comparison: 


We  have  tried  to  use  quantitative  measures  in  the  past 
to  compare  ourselves  to  the  competition  —  percent  of 
sales,  profit,  growth  and  all  that  —  which  really  was 
meaningless.   Then  we  said,  "Okay,  what  do  we  know 
about  what  our  competition  is  doing?'   And  we  went 
around  and  talked  to  our  business  managers  and  asked 
them  for  this  information.   It  is  available.   We 
talked  to  70  business  units.   Then  we  prepared  a 
qualitative  summary  that  explained  where  we  stood 
against  12  competitors  on  telecommunications, 
marketing  systems,  component  by  component.   We  tried 
to  do  it  in  a  way  that  would  give  top  management  a 
feel  for  where  we  are  ahead,  where  we  are  comparaole, 
and  where  we  are  behind.  We  also  put  the  whole  cake 
back  together  and  subdivided  it  into  major  functions 

—  marketing,  manufacturing,  research  and  development 

—  and  tried  to  figure  out  where  we  stand  in  basic 
computer  technology.   We  used  a  number  of  consultants 
and  tried  to  get  as  many  inputs  as  we  could.   Our 
executives  were  more  interested  in  the  qualitative 
understanding  than  they  had  been  in  the  quantitative 
approach. 

(Manager) 


Expert  opinion  is  divided  on  whether  spending  levels  are  a  meaningful 
output  measure  for  competitive  performance  assessment.   For  example,  in 
the  banking  industry,  consulting  firms  widely  publish  for  all  major 
competitors  the  ratios  of  I.T.  spending  compared  to  non-interest 
expense.  Among  this  panel  of  experts,  there  was  at  least  one  critic  of 
using  such  a  measure.   He  offered  a  cautious  note: 

I  don't  have  a  consensus  from  my  clients  or  colleagues 
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on  this,  but,  in  my  view,  the  average  standards  for 
determining  what's  worth  doing  in  a  business  in  terms 
of  new  investments  are  uniforml/  high.   I  believe 
everyone  is  over-invested  because  I  believe  there's  a 
lot  of  junk  out  there. 

(Consultant ) 


The  next  two  methods  are  particularly  useful  for  addressing  the  CEOs' 
question,  "Are  we  doing  the  right  things?",  because  each  employs  multiple 
assessment  methods,  as  well  as  behavioral  and  output  variables. 


Assessing  Business  Alignment 

The  business  alignment  assessment  examines  the  criticality  of  a  firm's 
operating  systems  and  its  portfolio  of  applications  to  its  business 
strategy.   This  is  accompanied  by  asking  the  following  set  of  questions: 

1.  What  makes  this  business  successful? 
What  does  it  need  to  do  to  compete? 
(Similar  to  a  critical  success  factors 
approach. ) 

2.  What  is  being  spent  on  I.T.?  What  is 
the  strategic  plan  for  I.T.?  What  is 
the  age  of  the  backlog  of  existing 
systems  or  the  installed  base?  What 
has  been  the  budget  growth  of  I.T.? 

3.  How  do  I.T.  expenditures  and  activities 
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map  onto  business  activities? 

4.  Is  the  r.r.  function  integrated  with 
the  business  units?   Is  the  I.T.  group 
responding  to  the  business  strategy  and 
having  an  impact  on  the  business  units? 

5.  What  are  the  implications  o£  a 
disruption  of  I.T.  service  to  the 
business  units? 

6.  How  does  this  alignment  compare  to  the 
competition's?   How  critical  is  I.T.  as 
a  factor  of  competition?   Has  the 
company  made  a  successful  "competitive 
move"  with  I.X.? 

Based  on  this  systematic  analysis,  the  firm  then  is  typed  according  to 
the  criticality  of  I.T.  to  the  achievement  of  its  business  objectives. 
In  the  last  step  of  this  assessment,  the  firm's  level  of  investment  in 
I.T.  is  compared  to  its  competition. 

Although  this  method  relies  on  some  of  the  same  indicators  as  the  other 
methods  discussed  so  far,  it  is  grounded  in  some  fundamentally  different 
assumptions.   First  and  foremost  is  that  it  takes  into  careful 
consideration  not  only  the  business  goals  but  the  constraints  under  which 
the  firm  operates.   It  assumes  that  the  criticality  and  strategic 
importance  of  I.T.  to  a  particular  firm  may  change  over  time,  due  to 
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competitive  actions  and  other  changes  in  the  external  environment,  or  to 
internal  factors,  such  as  learning  and  adaptation.   One  expert  who  uses 
this  method  commented: 


Contigent  models  are  critical.   You've  got  to  figure 
out  if  it's  a  matter  of  life  or  death.   You  know,  they 
nearly  lost  the  bank  at  one  of  the  leading  investment 
banks  this  summer  with  a  flood  in  the  computer  room. 
That  kind  of  company  thinks  about  it  differently  than, 
say,  a  utility  where  their  big  application  is  the 
weekly  billing  cycle  and  disruptions  are  irritating 
but  not  fatal.   As  you  move  from  the  utility  to  the 
investment  bank,  you  use  very  different  kinds  of 
measures  of  effectiveness  in  assessing  the  operation. 

(Professor) 


This  method  incorporates:  1)  an  instrumental  test  which  examines  the 
contribution  of  I.T.  to  the  ousiness  goals;  2)  a  social  test  of  the 
extent  to  which  business  unit  activities  and  I.T.  activities  are 
integrated;  and  3)  a  test  of  social  comparison  of  spending  levels  and 
criticality  of  individual  systems  with  comparable  organizations.   By 
employing  multiple  measures  and  methods,  managers  may  expand  their 
ability  for  addressing  the  question  "Are  we  doing  the  right  things?" 


Assessing  Investment  Targeting 

The  next  method  might  be  called  Investment  Targeting  Assessment.  One 
expert  asserts: 

I  maintain  that  if  your  investment  in  I.T.  is  paying 
off,  then  it  will  show  up  somewhere  in  expenses  and 
profits.   If  you  cannot  see  it,  then  it  does  not 
exist.   The  argument  is  that  your  total  investment  in 
I.T.  has  to  impact  your  cost  structure,  your  revenue 
structure  or  the  investment  base. 

(Consultant) 
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An  underlying  assumption  of  this  method  is  that  a  firm's  pattern  of 
spending  on  I.T.  is  its  de  facto  I.T.  strategy.   Undertaking  such  an 
assessment  involves  the  following  steps: 

Step  1:    Segmenting  all  users  into  functional  areas  and 

segmenting  major  categories  of  information  technology. 

Step  2:  Comparing  gross  average  spending  levels  for  each 
segment  to  competitors'  spending  levels  or  to  an 
industry  standard. 

Step  3:    Indentifying  business  objectives.   (Again,  a  critical 
success  factors  approach  may  be  used). 

Step  4;    Identifying  organization  pressure  points.   These  are 

the  critical  variables  that  yield  the  highest  leverage 
for  achieving  business  objectives  and  can  be  described 
in  terms  of:  1.  priority  functions;  2.  priority  job 
families;  3.  priority  processes;  and  4.  priority 
products. 

Step  5:     Using  the  segmentation  scheme  developed  in  Step  1, 
identify  the  investments  in  I.T.  that  will  have  the 
greatest  impact  on  the  organization  pressure  points. 

Step  6:     Comparing  pressure  points  and  key  investment  targets  to 
competitors. 
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Like  the  business  alignment  assessment,  this  approach  incorporates 
multiple  methods,  although  it  does  not  include  any  behavioral  variables 
of  the  kind  used  in  the  first  t/pe  of  social  test.   Included  are  a  social 
test  of  spending  comparability  and  a  comprehensive  instrumental  test  in 
which  careful  prioritization  of  functional  outputs,  human  resource 
outputs,  and  product  outputs  are  made.   This  method  would  be  appropriate 
when  knowledge  of  the  transformation  process  is  imperfect  and  output 
measurability  high. 


Assessing  Management  Vision 

The  seventh  and  last  expert  method  employs  measures  of  values,  attitudes, 
and  motivation  as  indicators  of  performance.   The  focus  is  on  top 
management's  understanding  of  the  strategic  value  of  information 
technology  and  their  ability  to  provide  leadership  and  direction  for 
future  action.   This  approach  to  evaluation  might  be  called  the 
management  vision  assessment.   For  many  of  the  experts  interviewed,  this 
indicator  of  effectiveness  is  perceived  as  one  of  the  most  important. 
One  expert  explained  the  problem: 


We  are  up  against  a  barrier  today.   We  brought  the 
technology  in  stand-alone  islands  of  automation  and  we 
cannot  move  forward  because  we  cannot  integrate  the 
pieces.   Secondly,  we  cannot  change  the  attitudes  and 
the  underlying  processes  that  really  open  the  door  to 
big  payoffs. 

(Consultant) 


The  expert  emphasized  the  need  for  a  new  type  of  I.T.  manager  with  values 
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distinctly  different  from  those  of  previous  eras.   One  expert  seemed  to 
speak  for  all  the  others  in  stating  that  the  top  I.r.  manager  mast  "speak 
the  language  of  the  general  manager": 


5fou  need  people  who  can  communicate.   People  who  feel 
comfortable  talking  to  senior  management.   People  who 
can  translate  problems  into  things  that  businessmen  can 
understand.   You  don't  want  to  tell  them  that  your 
operating  system  went  down  last  night  because  you  had  a 
SOC  7'.  What  do  they  care?  You've  got  to  put  it  in 
terms  that  they  can  understand. 

(Consultant) 


Another  stressed  the  importance  of  being  able  to  think  strategically: 


CIOs  [Chief  Information  Officers]  have  begun  to  play 
their  most  important  role  when  they  don't  view  their  job 
as  that  of  running  daily  operations,  but  as  managing 
long  range  strategic  thinking  and  architecture;  seeing 
that  you're  supporting  the  corporation  appropriately, 
that  you're  creating  a  techno-structure  that  is  going  to 
be  responsive  to  future  changes. 

(Consultant) 


Summarizing  for  the  other  experts,  one  of  the  consultants  said: 


One  thing  you  look  for  is  leadership  skills  in  the  head 
of  the  I.T.  function;  a  business  orientation  and 
leadership  skills  and  someone  whose  sense  of  values  has 
shifted  towards  business  values.   That's  critically 
important. 

(Consultant) 


The  search  and  recruitment  of  managers  with  such  values  is  one  indicator 
of  the  fundamental  transition  which  the  professional  managment  of  I.T. 
has  undergone  over  the  last  ten  years.   It  is  not  only  the  I.T. 
professional  whose  values  and  attitudes  have  changed,  however.   Many  of 
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these  experts  believe  that,  in  the  Euture,  it  v^ill  be  those  organizations 
whose  executives  have  fundamentally  different  values  and  attitudes  about 
I.T.  from  those  whose  visions  are  to  automate  who  will  be  most 
competitive.   One  expert  expressed  it  this  way: 


If  an  organization  says  that  it's  using  technology 
strategically  and  I  probe  what  that  means  in  terms  of 
organizational  change,  and  the  answer  comes  back,  "No 
problem,  we're  basically  going  to  automate  the  way  we've 
been  doing  things",  then  a  red  flag  goes  up.   The  basic 
position  I  take  is  that  if  it's  not  changing  the 
organization,  then  it's  having  a  marginal  effect.   I 
look  for,  on  the  line  side,  whether  they  have  linked 
their  notion  of  effective  use  of  technology  to  their 
notion  of  restructuring  the  organization. 

(Professor) 


And  perhaps  this  expect  opinion  was  the  most  critical; 


Very  few  managers  really  understand  what  technology  is, 
or  how  it  is  used  in  their  organizations. 

(Researcher) 


Another  expert  offered  his  perception  of  what  makes  a  "highly  aware 
manager": 


He's  the  one  who  always  sees  I.r.  rooted  in  a  business 
context.   Not  separate  and  distinct  from.   A 
right-thinking  senior  manager  ought  to  be  thinking  that 
an  effective  I.T.  function  is  one  which  both  figures  out 
how  to  find  the  real  high  value,  high  payback  uses  of 
information  technology,  and  does  it,  and,  at  the  same 
time,  is  more  selective  and  puts  tighter  filters  down  on 
the  everyday  run-of-the-mill  request  for  technology. 

(Consultant) 


One  expert  evoked  the  days  when  Walter  Wriston  was  the  chairman  of 
Citicorp: 
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One  measure  of  eEfectiveness  would  be  the  extent  to 
which  I.r.  has  been  made  part  of  everyone's  problem.   I 
think  that's  one  of  the  masterful  things  that  Citibank 
and  Wriston  did  back  in  the  1970's  when  the/  used  the 
term  electronic  banking.   What  they  meant  was  bankers 
who  understand  electronics.   Wriston  made  electronics 
the  job  of  the  banker,  not  the  other  way  around. 

(Consultant) 


Another  expert  characterizes  senior  managers  as  having  three  distinctly 
different  attitudes  and  behavior: 


The  first  type  is  the  committed  visionary  who,  given 
enough  information,  will  support  investments  in  new 
products  until  the  end.   If  others  are  disposed  to  get 
something  done,  this  manager  will  give  them  money  and 
support  and  will  stay  with  it  all  the  way  through. 
That's  the  committed  visionary. 

Then  there's  the  naive  visionary  who  hangs  up  when  you 
really  talk  about  change  in  the  business.   There's  the 
■I'll  take  six  pounds  of  American  Hospital  Supply" 
visionary  for  whom  technology  sounds  good.   "We  should 
be  modern.   Let's  have  some."   And  they  are  not  prepared 
for  changes. 

And  then  there  are  managers  who  are  about  the  making  of 
change.   They  didn't  come  to  technology  as  a  starting 
point.   They  may  or  may  not  understand  technology  well 
but  they  get  practical  and  very  tactical  about  getting 
it  done.  They  can  get  visionary,  but  it's  much  more  out 
of  a  business  context.   They're  the  best.   They  get  the 
job  done. 

(Consultant) 


Most  experts  expect  a  firm's  executive  committee  to  have  developed  a 
solid  understanding  of  I.T.'s  value  and  would  ask,  "Do  the  top  executives 
have  an  explicit  model  of  how  they  want  to  value  I.T.?"  Several  others 
would  want  to  know,  "Do  they  have  a  vision  of  what  will  be  required  in 
their  future  I.T.  infrastructure  in  order  to  support  changes  in  business 
strategy?"   Another  indicator  frequently  mentioned  is  whether  management 
is  knowledgeable  and  informed  about  the  new  technological  developments 
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that  ma/  represent  important  business  opportunities  in  the  future. 

There  is  an  implication  that  the  CEO  should  be  assessing 
someone  else  or  something  else ^_ily.  belief  i.s  that  it  is 
the  CEO's  responsibility  to  see  that- an  organization  is 
using  I.T.  to  its  greatest  advantage,  as  tt"would  be  for 
any  asset.   In  this  case,  we  happen  to  be  talking  about 
an  asset  that  is  changing  the  structure  of  industry  and 
society  around  us.   Change  creates  opportunity;  hence, 
executives  have  to  be  the  agents  of  change.   I  don't 
think  it's  someone  else's  responsibility;  it's  the  CEO's 
responsibility. 

(Consultant) 


And  so  we  have  come  full  circle,  from  the  uneasy  CEO  who  confessed  he  was 
unable  to  assess  I.T.  performance  to  the  expert  who  is  critical  of  any 
CEO  that  has  not  picked  up  the  gauntlet  of  strategic  change  and  I.r. 
vision.   What  dialectic  will  reconcile  these  perspectives? 


Conclusions 

The  problem  may  require  a  diagnosis  that  goes  beyond  the  usual 
prescriptions  of  "improve  communications  between  the  technologist  and 
executives,"  "develop  a  strategic  I.T.  plan,"  or  "form  an  I.T.  steering 
committee."   Instead,  what  may  be  required  is  a  radical  rethinking  of  the 
process  for  formulating  expectations  and  assessing  performance. 

In  figure  5,  each  of  the  seven  expert  methods  is  analyzed  in  terms  of  its 
appropriateness  for  different  conditions. 
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Figure  5.   METHODS  FOR  ASSESSING  I.T.  PERFORMANCE 
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For  executives  with  visions  to  automate  their  organizations,  I.r. 
performance  should  be  evaluated  with  productivit/  methods,  and  the 
emphasis  should  be  on  identifying  indisputable  and  clear  standards  of 
performance.   But,  in  situations  where  knowledge  about  the  transformation 
process  is  imperfect  and  performance  standards  are  not  clear,  such  as 
those  in  which  a  vision  to  informate  or  transform  the  organization 
dominates  organizational  action,  the  use  of  multiple  methods,  such  as 
competitive  performance  assessment,  business  alignment  assessment,  or 
management  vision  assessment,  are  more  appropriate.   We  hypothesize  that 
the  control  systems  in  many  companies  which  were  developed  for  an 
automating  vision  of  I.T.  do  not  provide  the  type  of  information 
executives  need  for  determining  whether  they  are  fit  for  future  action. 
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Top  executives  often  expect  performance  to  be  evaluated  with  simple, 
quantifiable  output  measures  because  the/  are  not  knowledgeable  or 
familiar  with  I.r.   But  instrumental  tests  of  what  is  good  performance 
can  be  extremely  unstable  because  they  are  built  on  assumptions  that  past 
performance  is  a  reliable  means  for  predicting  future  performance. 

We  lack  the  metrics  for  assessing  performance  for  visions  to  transform  or 
informate  organizations.   An  assessment  for  an  informating  vision  relies 
primarily  on  behavioral  variables:   changes  in  patterns  of  information 
use,  changes  in  roles  and  relationships  between  managers  and  employees, 
more  comprehensive  jobs,  and  changes  in  the  fundamental  premise  of 
organizational  authority. 

Direct,  expert  and  objective  observation  is  required  to  make  such 
judgements.  One  expert  explained  that  these  expectations  require 
assessments  that  are  more  behaviorally  focussed: 


By  interviewing  everyone  exposed  to  the  technology,  I 
have  a  much  clearer  and  more  objective  picture  of  what 
is  actually  going  on  than  any  one  individual  does.   The 
picture  I  would  draw  would  be  much  more  than  what  any 
one  indiviual  understood.   The  data  are  not  only 
informing,  but  they  become  an  artifact  of  organizational 
life. 

People  have  limited  cognitive  maps  not  only  of  what 
they're  doing  with  technology  but  of  what  they  could  be 
doing.   This  limited  cognitive  map  is  a  consequence  of 
having  no  time  for  reflecting  and  understanding 
technology.  Many  want  assessment  methods  that  are  mass 
produceable.  This  is  poor  social  science  and  reflects 
the  worst  of  scientism.   What  I'm  suggesting  is  that 
finding  ways  of  pointing  out  inconsistencies  in  values 
and  behavior  is  a  valuable  process  for  answering  the 
question  "Are  we  using  I.T.  effectively?" 

(Researcher) 
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This  ma/  explain  wh/  man/  senior  managers  are  dissatisfied  »/ith 
traditional  measures  of  I.r.  productivity.   In  the  words  of  one  expert; 


Everything  is  cleaner  to  assess  at  the  individual  system 
level.   But  it  is  a  question  of  "Is  your  I.T. 
infrastructure  merely  the  summation  of  your 
applications?"   My  answer  to  that  is  no.   There  is  more 
going  on.   There's  this  whole  strategic  piece.   This 
asks,  "How  well  positioned  am  I  to  solve  tomorrow's 
problems  with  information  technology?" 

(Consultant) 


l£  top  management  is  explicit  about  their  expectations  for  transforming 
the  organization,  this  places  demands  on  every  business  and  functional 
manager  to  search  actively  for  radical  new  product  and  service  ideas,  and 
to  establish  extremely  ambitious  performance  targets.   Such  a  vision 
fosters  higher  than  normal  levels  of  risk  taking  behavior.   Developing 
and  employing  reliable  measures  and  assessment  procedures  can  be  costly, 
and  managers  may  avoid  them  because  of  these  costs.   But  if  managers 
rafuse  to  reexamine  whether,  in  fact,  their  control  systems  are 
appropriate  for  their  expectations  of  information  technology,  they  run 
the  risk  of  being  seduced  by  reports  of  performance  based  on  measures  and 
methods  that  are  not  suitable  for  answering  the  CEO's  question,  "Are  we 
fit  for  future  action?" 

In  closing,  for  managers  who  have  concerns  about  how  they  assess  I.T. 
performance,  we  would  like  to  suggest  several  ideas.   First,  conduct  an 
audit  of  all  the  various  assessment  approaches  currently  being  used  and 
then  categorize  them  in  terms  of  whether  they  rely  on  output,  behavioral 
or  attitudinal  variables.   With  the  control  framework  presented  here, 
analyze  the  appropriateness  of  each  for  different  conditions.   Interview 
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ke/  executives  and  users  to  find  out  how  comfortable  they  are  with  these 
control  systems.   Then  analyze  what  are  your  managers'  visions  for  I.T. 
at  different  levels  of  the  organization  and  across  different  business  or 
functional  units.   Finally,  reevaluate  the  appropriateness  of  youc 
control  systems  in  context  of  top  management's  expectations  for 
information  technology.   All  I.r.  professionals  evaluate  what  they  are 
doing,  but  few  have  a  rational  structure  for  doing  it.   A  firm's 
assessment  process  is  a  primary  mechanism  for  learning  and  adaptation, 
and  it  should  incorporate  more  than  just  one  method. 
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NOTES 


1.  Schein,  Edgar  H.  and  Diane  D.  Wilson,  "Executive  Managements' 
Explicit  Assumptions  about  Information  Technology  and  Its  Role  in  their 
Organizations",  Unpublished  report  to  the  M.I.T.  Management  in  the  1990s 
Steering  Committee,  Summer,  1987. 

2.  ZuboEf,  Shoshona,  In  the  Age  of  the  Smart  Machine;  The  Future  o£ 
Power  and  Work,  Basic  Books,  New  York,  1988,  p.  9. 

3.  Personal  conversation  with  Edgar  Schein,  December,  1987. 

4.  Zuboff,  p.  9. 

5.  In  Organizations  in  Action  (1957),  Thompson  first  defines  efficiency, 
instrumental  and  social  tests,  and  the  conditions  which  determine  their 
appropriateness.   His  ideas  about  social  tests  were  derived  from 
Festinger's  theory  o£  social  comparison  processes.   Ouchi  and  Maguire 
(1975)  applied  these  constructs  to  a  study  of  197  departments  in  5  retail 
department  stores  and  concluded  that  output  measures  are  used  most  when 
least  appropriate,  under  conditions  of  complexity,  interdependence  and 
lack  of  expertise,  and  that  organizations  should  use  two  forms  of 
control,  one  to  serve  institutional  needs  and  one  to  serve  subunit 
needs.   Ouchi  (1979)  and  Eisenhardt  (1985)  further  developed  Thompson's 
criteria  for  determining  appropriate  control  systems. 

6.  Porter,  Michael  E.  and  Victor  E.  Millar,  "How  Information  Gives  You 
Competitive  Advantage",  Harvard  Business  Review,  July-August,  1935.  p. 
151. 

7.  Rockart,  John  F.,  "Chief  Executives  Define  Their  Own  Data  Needs", 
Harvard  Business  Review,  March-April,  1979,  p.  81. 
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